Motivation

I've been researching LLM-agent-based usability testing recently, and given how capable local open-weight models have become, it seemed like UXAgent could realistically run on fully local models — no cloud API keys, easier to iterate on experiments, and lower cost for large batch runs. Since the docstring in async_chat already hints at adding custom providers through the LiteLLM routers, I followed that path and added first-class support for Ollama and vLLM.

Changes

• agent/gpt.py
  • Added ollama and vllm entries (plus _thinking variants) to the chat, slow-chat, and embedding LiteLLM routers. Models and endpoints are configurable via environment variables (OLLAMA_CHAT_MODEL, OLLAMA_SLOW_CHAT_MODEL, OLLAMA_EMBEDDING_MODEL, VLLM_API_BASE, VLLM_CHAT_MODEL, …) with sensible defaults.
  • Moved load_dotenv() before router construction — previously it ran after the routers were built, so .env settings never applied to them.
  • Enabled native JSON response_format for the ollama/vllm providers (both backends support it, and it significantly improves JSON reliability for local models). The existing regex-extraction fallback is unchanged.
  • Added an optional LLM_MAX_TOKENS env cap: async_chat defaults to max_tokens=64000, which vLLM rejects with a 400 when it exceeds the served model's context length.
  • Added an optional EMBEDDING_PROVIDER env override so chat and embeddings can use different providers (a vLLM server typically serves a single model, so e.g. vLLM chat + Ollama embeddings).
• conf/base.yaml: documented the new provider options for llm_provider.
• README.md: added a "Using local models (Ollama / vLLM)" section, including the Ollama context-window caveat and notes on model-size requirements.

No agent logic was touched, and the three existing cloud providers (openai / aws / anthropic) behave exactly as before.

Testing

Verified end-to-end with all cloud API keys removed from the environment:

• Ollama (llama3.1 + nomic-embed-text): async_chat (fast + slow routers), json_mode, sync chat, and embed_text all pass.
• vLLM code path (hosted_vllm/) validated against an OpenAI-compatible endpoint, including the LLM_MAX_TOKENS cap and JSON mode.
• EMBEDDING_PROVIDER override (vllm chat + ollama embeddings) works.
• ruff check passes.

🤖 Generated with Claude Code